This application relates to super resolution (SR) image construction from low resolution image(s).
Most of the visual data used by digital imaging and display devices are represented as raster image with a rectangular grid of pixels. Since images with higher pixel density are desirable in many applications, there is an increasing demand to acquire high resolution (HR) raster images from low resolution (LR) inputs such as images taken by cell phone cameras, or converting existing standard definition footage into high definition image materials. However, raster images are resolution dependent, and thus cannot scale to an arbitrary resolution without loss of apparent quality. The performance of simple interpolation methods is limited because they poorly recover the missing high-frequency components, and often blur the discontinuities of magnified images.
While the raster image representation is resolution dependent, vector image is a resolution-independent way to represent visual data digitally. The vector representation is scalable because it represents the visual data using geometrical primitives such as points, lines, curves, and shapes or polygon. This capability largely contrasts the deficiency of raster representation. The idea of automatically converting raster image into more scalable vector representation has been studied. However, vector-based techniques are limited in the visual complexity and robustness. For real images with fine texture or smooth shading, these approaches tend to produce over-segmented vector representations using a large number of irregular regions with flat colors. The discontinuity artifacts in region boundaries can be easily observed, and the over-smoothed texture regions make the scaled image watercolor like.
Images have been encoded using effective linear transforms. In this way the rich visual information can be encoded into the applied transformation, and the image is vectorized as the coefficients of a small set of bases of the transformation. In simple scenarios, pre-fixed transformations can be used, such as the BiLinear/BiCubic basis functions in 2-D image interpolation, and the DCT/DWT adopted in JPEG/JPEG-2000 standard, among others. Anisotropic bases such as countourlets have been explicitly proposed to capture different 2-D edge/texture patterns. These techniques usually lead to sparse representation which is very preferable in image/video compression. In addition to pre-fixed bases, since natural images typically contain many heterogeneous regions with significantly different geometric structures and/or statistical characteristics, adaptive mixture model representations were also reported. For example, the Bandelets model partitions an image into squared regions according to local geometric flows, and represents each region by warped wavelet bases; the primal sketch model detects the high entropy regions in the image through a sketching pursuit process, and encodes them with multiple Markov random fields. These adaptive representations capture the stochastic image generating process, therefore they are suited for image parsing, recognition and synthesis.
Simple interpolation methods cannot recover the missing high-frequency components and often blur the discontinuities of magnified images. The super-resolution (SR) approach reconstructs the high-resolution (HR) image by recovering additional information from multiple LR images using signal processing techniques. Existing SR methods can be divided into three major categories, specifically the functional-interpolation, the reconstruction-based and the learning-based methods. Comparing to other two categories, the learning-based methods are able to process single image input, and are more stable by using statistical learning technique, hence they have become a promising direction for the SR problem.
The learning-based methods solves the SR problem by learning the co-occurrence prior between HR and low-resolution (LR) image patches or coefficients, and processing the LR input along with appropriate smoothness constraint to generate HR image. However, in typical Learning-based image SR approaches, different models are built for different scale factors, and usually these models are not interchangeable. For example, the model computed for ×2 scale SR cannot be used for ×3 SR. These systems ignore the relation between models of different scales, hence large storage space and computation resources were wasted, and these techniques are quite limited in their applications.